Vehicle imaging system with stereo imaging

ABSTRACT

A vehicle imaging system includes an imaging array sensor having a field of view directed outwardly from the vehicle and toward an external scene, first and second optic elements, and a control responsive to an output of the imaging array sensor. The imaging array sensor has at least first and second portions, with the first optic element positioned along a first optic path between the first portion of the imaging array sensor and the external scene, and the second optic element positioned along a second optic path between the second portion of the imaging array sensor and the external scene. The control processes the output and distinguishes the first image and the second image in order to identify objects of interest in the external scene. The control generates a control output in response to the processing.

This application is a continuation-in-part of co-pending U.S. patentapplication Ser. No. 09/313,139, filed on May 17, 1999, which is acontinuation of application Ser. No. 08/935,336, filed on Sep. 22, 1997,which is a continuation of U.S. Pat. No. 5,670,935, filed on May 22,1995, the disclosures of which are hereby incorporated herein byreference.

BACKGROUND OF THE INVENTION

This invention relates generally to vehicular vision systems and, moreparticularly, to a vehicular vision system which is operable todetermine a distance from the vehicle to an object or light sourceremote from the vehicle. More particularly, the present invention isdirected to determining the distance to an object whose image iscaptured by an image capture device. One application for the imagingsystem of the present invention is with a vehicle headlamp control andmay identify particular light sources of interest and adjust a vehicle'sheadlamps according to the distance between the vehicle and theparticular light sources.

Vehicle camera or vision systems have been proposed for variousapplications, such as rear and/or side view vision systems, back upaids, collision avoidance systems, rain sensor systems, head lampcontrol systems and the like. These systems may include a camera orsensor positioned on the vehicle for capturing an image of a sceneexteriorly of the vehicle. The vision systems may also include a displayfor displaying a captured image, or may control an associated accessoryon the vehicle, such as windshield wipers, headlamps or even the brakesystem in response to one or more characteristics of the captured image.In some applications, it has been recognized that distance informationbetween the vehicle and an object in the captured scene may be helpful.In such applications, a ranging device may also be included to providethis information. Various ranging devices have been proposed, such asradar, ultrasonic, sonar, infrared beam/detector devices or similarproximity sensing devices. While such devices provide distanceinformation to the associated vehicular system, this requires anadditional sensing device separate from the vehicular vision or camerasystem, which adds to the bulk and costs associated with the system.

One vehicle system which distance information may be particularly usefulis a vehicle headlamp control system for adjusting a vehicle headlamp inresponse to a detection of oncoming headlamps or leading taillightsassociated with other vehicles. To date, there have been many proposedheadlight dimmer control systems. Many of the prior attempts at vehicleheadlight dimming controls include a single light sensor whichintegrates light from a scene remote from the vehicle. The vehicleheadlights are then dimmed when the integrated light exceeds apredetermined threshold. However, these systems typically require asufficiently low threshold of detection such that many other lowerintensity light sources may also be interpreted as headlights ortaillights. These systems also have difficulties in reliably detectingtaillights of other vehicles traveling ahead of the operative vehicle,since the intensity of taillights is typically substantially less thanthe intensity of oncoming headlights.

Other proposed headlight dimming controls implement an imaging arraysensor which not only senses the light originating from both headlightsand taillights, but may further determine the color and intensity of thelight, thereby further determining whether the light source is aheadlight or a taillight. Such systems are deficient in determining thedistance between the sensed light source and the subject vehicle, whichwould be helpful modulating the headlamps in response to both the sensedlight and the distance to the light. One proposed solution is toestimate the distance between the vehicle and the light source inresponse to the brightness or intensity of the sensed light source,since the detected signal from the light source may at times vary withthe square of the distance to the light source. However, such acalculation is only accurate when the sensed light source intensity iswithin a predetermined level corresponding to a known or assumedintensity of headlamps and is at certain distances. Because theintensity of headlamps and taillamps vary between vehicles and mayfurther vary as the headlamps are modulated between high and low beamsand as the brake lights are activated or deactivated, such an estimationof distance may be inaccurate in many cases.

SUMMARY OF THE INVENTION

The present invention provides a vehicular imaging system which iscapable of accurately determining the distance from the subject vehicleto an object or light source sensed by the sensors of the imagingsystem. The distance sensor accurately estimates the distance betweenthe sensed object and the vehicle, while avoiding excessive additionalcosts and bulk to the vehicle vision and/or control system. In oneaspect, the present invention is intended to provide a vehicularheadlamp control system which senses oncoming headlights and leadingtaillights of other vehicles and controls the headlamps of the subjectvehicle in response to the sensed light sources and the distance betweenthe vehicle and the sensed light sources. The control system preferablyincludes ranging capability for determining the distance between thesensed objects and the vehicle. The device preferably is adaptable foruse in other vehicular imaging systems associated with the vehicle whichmay display a distance readout to an operator of the vehicle or maycontrol a vehicle accessory in response to the distance.

According to an aspect of the present invention, a vehicular imagingsystem comprises at least one imaging array sensor and a control. Theimaging sensor is mounted at a vehicle and has stereoscopicdistance-sensing capability. The control is responsive to an output ofthe imaging array sensor in order to capture an image of at least oneobject external of the vehicle and determine a distance between theimaging array sensor and the object.

Preferably, the imaging array sensor receives a stereoscopic image of ascene remote from the imaging array sensor. The stereoscopic imageincludes a first image of an object in the scene on a first portion ofthe imaging array sensor and a second image of the object on a secondportion of the imaging array sensor. The control is responsive to theimaging array sensor in order to determine a distance between theimaging array sensor and the object.

In one form, the vehicular imaging system is implemented in a vehicularheadlamp control system, such that the headlamps are modulated betweenhigh and low beams in response to the distance between the sensed objector light source, which may be representative of an oncoming headlight orleading taillight, and the imaging array sensor.

In another form, the vehicular imaging system includes first and secondimaging array sensors such that the first image of the object isreceived by the first imaging array sensor and the second image of theobject is received by the second imaging array sensor. Preferably, afirst and second optic element is included along the respective opticpaths between the first and second imaging array sensors and the scene.The distance between the object and the sensors may then be determinedas a function of a relative position of the image of the object asreceived on the first and second imaging array sensors and the focallengths of the first and second optic elements.

According to another aspect of the present invention, a vehicularheadlamp control for modulating a headlamp of a vehicle comprises atleast one imaging array sensor adaptable to receive a stereoscopic imageof a scene remote from the vehicle and a control responsive to theimaging array sensor. The imaging array sensor receives a plurality ofimages associated with a plurality of light sources associated with thescene. The control identifies light sources of interest and provides acontrol output to the vehicle. The control calculates a distance betweenat least one of the light sources and the imaging array sensor andprovides the control output in response to the distance. The headlampcontrol modulates the headlamps of the vehicle in response to thecontrol output.

According to another aspect of the present invention, a rearview visionsystem for a vehicle comprises at least one imaging array sensor and acontrol. The imaging array sensor is positioned on the vehicle and isdirected outwardly from the vehicle. The imaging array sensor hasstereoscopic distance-sensing capability. The control is operable todetermine a distance from at least one object exteriorly of the vehiclein response to an output of the imaging array sensor.

These and other objects, advantages, purposes and features of thisinvention will become apparent upon review of the followingspecification in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a plan view of a vehicle incorporating the present invention;

FIG. 2 is a block diagram of the imaging system of the presentinvention;

FIG. 3 is a block diagram of an imaging sensor useful with the presentinvention;

FIG. 4 is a schematic diagram of a light-sensing array useful with thepresent invention;

FIG. 5 is the same view as FIG. 3 illustrating the geometricrelationship between an object and the imaging sensor useful with thepresent invention;

FIG. 6 is the same view as FIG. 4, with shading of the pixels indicatingpixels sensing an object or light source;

FIG. 7 is the same view as FIG. 6 with similarly illuminated pixelsbeing designated as groups of pixels or segments;

FIG. 7A is a schematic of three-pixel sub-array useful for identifyingand labeling the segments illustrated in FIG. 7;

FIGS. 8A and 8B are the same view as FIG. 6 of first and second imagingarrays useful with the present invention, with the similarly illuminatedgroups of pixels being labeled as discreet groups or segments;

FIG. 9 is a flow-chart of a segment labeling process useful with thepresent invention;

FIG. 10 is a flow-chart of a process for determining the position andintensity of the segments;

FIG. 11 is a flow-chart of a process for determining whether aparticular segment on a first imaging array sensor is an image of thesame object as a corresponding segment on a second imaging array sensor;

FIG. 12 is a flow-chart of the stereoscopic distance determinationfunction of the present invention;

FIGS. 13A-C are schematics of various embodiments of a stereoscopicimaging sensor with distance determining capability within a housing,such as an interior rearview mirror assembly housing;

FIG. 14 is a side elevation of a portion of a vehicle embodying aheadlamp dimmer control in accordance with the present invention;

FIG. 15 is a partial side elevation view and block diagram of thevehicle headlight dimming control of FIG. 14;

FIGS. 16A and 16B are flow-charts of the stereoscopic headlamp controlprocesses in accordance with the present invention; and

FIGS. 17A-C are curves of segment intensity versus distance useful indetermining whether to activate or deactivate the high or low beams ofthe headlamps.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring now specifically to the drawings and the illustrativeembodiments depicted therein, a vehicle 10 includes a vehicle imagingsystem 12 which includes an imaging sensor module 14 and an imagingcontrol 16, as shown in FIGS. 1, 2 and 3. Vehicle imaging system 12 maybe a rearview vision system of the type disclosed in commonly assignedU.S. Pat. No. 5,670,935, a rearview vision system of the type disclosedin commonly assigned published PCT Application, InternationalPublication No. WO96/38319, published Dec. 5, 1996, a wide angle imagecapture system of the type disclosed in commonly assigned co-pendingU.S. patent application Ser. No. 09/199,907, filed Nov. 25, 1998 byBrent J. Bos, et al., a rain sensor and the like of the type disclosedin commonly assigned published PCT application, InternationalPublication No. WO 99/23828, published May 14, 1999, or a headlampdimming control of the type disclosed in U.S. Pat. No. 5,796,094, issuedto Schofield et al., the disclosures of which are hereby incorporatedherein by reference. Imaging sensor module 14 senses light from a sceneoutside of vehicle 10 and imaging control 16 receives an output fromsensor module 14. Imaging array module 14 is operable to facilitatedetermination of a distance between the module 14 and an object, such asa light source, in the target scene by receiving a stereoscopic image ofthe object on a pair of imaging sensors 34 a and 34 b or a dividedsensor. By comparing the relative locations or registrations of aparticular object or light source in the target scene on each of theimaging sensors 34 a and 34 b, the distance to the object may bedetermined as discussed below. Vehicle imaging system 12 may include adisplay 13 or other means for conveying the distance to an operator ofvehicle 10 or may respond to the distance determination by controllingan accessory or device such as a warning indicator or signaling deviceor even the brake system of the vehicle if the control is associatedwith a collision avoidance system or the windshield wipers and/orheadlamps if the control is associated with a rain sensor and/orheadlamp control, respectively. If associated with a headlamp control,the distance is used to detect when headlamps or taillamps are at adistance where the headlamps of the controlled vehicle should be dimmed.

As shown in FIG. 1, a backup aid or rear view vision system 70 may bepositioned on a rearward portion 72 of vehicle 10 and may comprise astereoscopic imaging system. Rear view vision system 70 may alternatelybe positioned on side rearview mirrors 70 a or on the rear view mirror30 within the vehicle. It is further envisioned that the imaging sensors34 a and 34 b may be integrally constructed to a housing or fixedportion of the bracket of the exterior mirror, thereby combining thesensors or cameras within the mirror to form a single unit. Thestereoscopic vision system may then determine the distance from thevehicle to an object rearward of the vehicle and provide a distanceoutput to an operator of vehicle 10. The vision system may include adisplay 13 which provides an operator of the vehicle with an image ofthe scene remote from the vehicle and a distance readout to an object orobjects in the scene.

Preferably, the image may be displayed as a unitary image synthesizedfrom outputs of two or more imaging sensors. Image enhancements may alsobe provided in the displayed image to further enhance the driver'sunderstanding of the area immediately surrounding vehicle 10. Forexample, graphic overlays, such as distance indicia in the form ofhorizontal grid markings or the like, may be provided to indicatedistances between the vehicle and objects displayed in display 13. Thesegraphic overlays may be superimposed on display 13 and thus are visibleto the operator of vehicle 10. The grid markings may be moved, curved orotherwise adjusted in response to a change in the vehicle's direction oftravel, which may be determined by a change in the vehicle's steeringsystem, the vehicle's differential system or a compass heading.Additionally, images of objects or other vehicles may be adjusted orenhanced in response to the distance between vehicle 10 and the othervehicles, such as by flashing or changing the color of images of objectswithin a threshold distance of vehicle 10. Alternatively, the distanceto multiple objects or a distance to a closest object may be displayedon display 13 or otherwise communicated to the vehicle operator. Thedistance to several objects may be displayed or the operator may selectone or more particular objects in the display for which the distance isdetermined. The selection may be made by a mouse, keypad, joystick orthe like.

Alternately, the stereoscopic vision system may be implemented with arain sensor 80, which may be placed inside the vehicle passengercompartment and directed toward a window or windshield 26. Rain sensor80 may then be operable to determine a distance from the sensor to thesensed droplets, in order to ensure that the sensed droplets arepositioned on the windshield 26 of vehicle 10 and not remotelypositioned therefrom, thereby reducing the possibility of a falsedetection of rain on the windshield.

As mentioned above, the stereoscopic imaging system is also useful witha vehicle headlamp dimming control 12′. The headlamp control 12′ may beimplemented in a rearview mirror assembly 30 and directed forwardly ofvehicle 10 (FIG. 14). Headlamp control 12′ may then adjust or modulatethe headlamps 20 of vehicle 10 in response to a distance between thevehicle and oncoming headlamps or leading taillights of other vehicles.This substantially reduces the possibility of modulating the headlampswhen the detected vehicle is substantially distant from vehicle 10.

Referring now to FIG. 3, imaging sensor module 14 preferably includes apair of imaging array sensors 34 a and 34 b, each of which receives animage of the target scene via a pair of focusing lenses 36 a and 36 band a pair of color filters 38 a and 38 b, respectively, all of whichare positionable along respective optic paths between the target sceneand imaging array sensors 34 a and 34 b. Control 16 receives an outputfrom each imaging array sensor 34 a and 34 b and converts the output todigital values via an analog to digital converter (not shown) andcommunicates the values to an appropriate control logic, such as avehicle lighting control logic module 18 (FIG. 15). Control 16 furtherfunctions to at least occasionally activate each imaging array sensor 34a and 34 b and analyze the output of each to determine the type of lightsource sensed and a distance from the vehicle to the light source.

Preferably, imaging arrays 34 a and 34 b are pixilated imaging arraysensors, such as a CCD or a CMOS sensor, although other array sensorsmay be implemented without affecting the scope of the present invention.As shown in FIG. 4, each of the imaging array sensors 34 a and 34 b arepreferably similar to the type disclosed in commonly assigned U.S. Pat.No. 5,550,677 issued to Kenneth Schofield and Mark Larson, thedisclosure of which is hereby incorporated herein by reference. Becausethe imaging array sensors are described in detail in the Schofield '677patent, the specific details will not be further discussed herein.Briefly, each of the imaging array sensors 34 a and 34 b preferablycomprise a plurality of photon accumulating light sensors or pixels 42.The array of photo-sensors 42 are interconnected to a vertical shiftregister 46 and a horizontal shift register 52 via a common word line 44and a common bit line 48, respectively. The bit lines 48 are alsointerconnected with amplifiers 50. The registers 46 and 52 function toindividually access each photo-sensor pixel or element 42 and provide anoutput 56 associated with the individual signals to the analog todigital converter of control 16.

As imaging array sensors 34 a and 34 b receive light from objects and/orlight sources in the target scene, control 16 may then be operable todetermine a color or other characteristic, such as intensity or size,being communicated by the sensed light sources, which may further bedetermined to be a desired target object, such as a headlamp ortaillight, as disclosed in the Schofield '094 patent. Color filters 38 aand 38 b may also be used to determine the color of other light sourcesas well. The color filters may be conventional mosaic filters or thelike or may be electro-optic filters of the type disclosed in commonlyassigned and co-pending U.S. provisional patent application Ser. No.60/135,657, filed on May 24, 1999 by Mark L. Larson and Brent J. Bos,the disclosure of which is hereby incorporated herein by reference. Byreceiving a stereoscopic image on sensors 34 such that one image isreceived on one array 34 a while a corresponding image is received onthe second array 34 b, the distance to an object in the target scene maythen be determined as a function of the locations of each sensed imagerelative to a respective reference location, such as a center point oraxis, of the corresponding imaging array sensors, the separationdistance of the two arrays and the focal length of the focusing lensesor optics. This distance may be calculated according to the followingequation: $\begin{matrix}{{D = \frac{\Delta\quad f_{1}f_{2}}{{f_{1}x_{D2}} - {f_{2}x_{D1}}}};} & (1)\end{matrix}$where, as represented in FIG. 4, D is the straight-line distance fromthe sensed object to a forward surface 36 c of optics 36 a and 36 b, Δis the lateral separation distance between a mid-point, axis or otherreference point associated with each sensor 34 a and 34 b, f ₁ is afocal length of the first optic 36 a, f ₂ is a focal length of thesecond optic 36 b, x _(D1) is a directed distance from a center axis 34c of the first sensor 34 a to the sensed image 34 d of the object O onsensor 34 a and x _(D2) is a corresponding directed distance from acenter axis 34 f of the second sensor 34 b to the sensed image 34 e ofthe object O on sensor 34 b. The directed distances x_(D1) and x_(D2)may be positive or negative values in accordance with the location wherethe sensed images 34 d and 34 e are detected by sensors 34 a and 34 b,respectively. For example, x_(D1) and x_(D2) may both be positive inFIG. 5, but one or both may be a negative value if the object O ispositioned relative to the optics and sensors such that one or bothsensed images 34 d and 34 e are received by sensors 34 a and 34 b on theother side of the center axes 34 c and 34 f, respectively.

Once the distance D is known, the lateral distance X to the object O mayalso be determined by the equation: $\begin{matrix}{X = {\frac{{Dx}_{D2}}{f_{2}}.}} & (2)\end{matrix}$Similarly, the angle from the vehicle to the object 0 may easily becalculated by taking the inverse tangent of the lateral distance Xdivided by the longitudinal distance D or of the image position X_(D2)divided by the focal length f₂. Control 16 may then determine if thesensed object or light source is within a predetermined tolerance bandof a targeted object or light source, such as a typical headlamp ortaillight, both in intensity and in location (lateral and longitudinaldistance) relative to vehicle 10. If the intensity and distance of thesignal is within the tolerance or threshold levels, the signal may bedetermined to be one of the targeted objects and imaging system 12 mayrespond accordingly. For example, if imaging system 12 is associatedwith a vehicle headlamp control, imaging system 12 may adjust theheadlamps 20 of vehicle 10 in response to a distance and angle betweenvehicle 10 and the detected headlamps and/or taillights of othervehicles.

Referring now to FIGS. 6 through 8, the following illustrates anddescribes the processes through which control 16 may determine thedistance between a light source or other sensed object and the vehicle10. As shown in FIG. 6, the arrays 35 a and 35 b of the respectiveimaging array sensors 34 a and 34 b include pixels 42, which sense lightvalues representative of light sources and other objects present in thetarget scene. Although shown as an array comprising an 8×8 array ofpixels, the array is shown here as a small array for purposes of clarityonly, since typical imaging array sensors useful with the presentinvention may comprise approximately 512×512 pixel arrays or more. Thepixels 42 are shown with shaded pixels 42 a representing sensed lightvalues which are greater than a pre-determined noise level associatedwith the array sensors 34 a and 34 b.

When operable, control 16 may shutter or open each of the imaging arraysensors 34 a and 34 b to collect the signals from the target scene oneach array 35 a and 35 b. After the signal has been received andcommunicated to control 16, control 16 may function to identify andclassify each of the pixels in accordance with their intensity and coloras determined by control 16 and pixel assignment with respect to colorfilters 38 a and 38 b. For example, white pixels may be identified andanalyzed to determine whether the white pixels are headlamps of oncomingvehicles, and then red pixels may be identified and analyzed todetermine whether the red pixels are tail lights of the leading vehiclestraveling in the same direction ahead of the subject vehicle 10.Clearly, however, the pixels may be classified and analyzed according toother colors or intensities for determining the distance to otherobjects or light sources within the targeted scene, without affectingthe scope of the present invention.

As shown in FIG. 7, similarly illuminated pixels, having a similar colorand/or intensity, are similarly classified, such as red or white, andare shown as pixels 42 b with an “x” through them. Not all of the shadedpixels 42 a in FIG. 6 are similarly classified in FIG. 7 because some ofthe shaded pixels 42 a may represent a light value above the noisethreshold but from a different colored light source. The similarlyclassified pixels 42 b may then be assigned a value of one or otherwiselabeled, while the other blank pixels 42 may be assigned a value ofzero, for the purpose of determining connected segments or groups ofpixels corresponding to each particular light source in the targetscene. This is preferably accomplished by activating a segmentation andlabeling algorithm or process 100 which determines which of theclassified pixels 42 b belongs to each particular segment or lightsource and labels each segment in numeric order. Each segment of pixelswithin a particular classification, such as white, red or other color,is thus labeled as a discreet segment from the other pixels or segmentsof pixels with the same classification. Labeling algorithm 100preferably analyzes each pixel and compares the assigned value (such asone or zero) of each pixel to one or more neighboring pixels. A set ofneighboring pixels is represented by a three-pixel window or sub-array43 (FIG. 7A) which may be applied to each of the imaging arrays 35 a and35 b. The sub-array 43 is preferably moved through the array, startingat an upper left corner and proceeding left to right and then downwarduntil each pixel in the array has been analyzed and compared to itsneighboring pixels.

As sub-array 43 moves through arrays 35, each pixel 42 and 42 b isindividually analyzed by a leading pixel window 43 a to determine if theindividual pixel has been assigned a value of one. If the pixel isassigned as one, each of the neighboring upper and left pixels are alsoanalyzed by an upper and left pixel window 43 b and 43 c, respectively,in order to determine if an individual pixel that is assigned a value ofone is connected with one or more previously analyzed pixels similarlyassigned a value of one. A labeling window or sub-array 44 then furtheranalyzes the individual pixel with a labeling pixel window 44 a and theupper and left adjacent pixels with labeling pixel windows 44 b and 44c, respectively. Labeling sub-array 44 determines and compares thedesignated segment number for each of the previously analyzedneighboring or adjacent pixels and labels the subject individual pixelaccordingly. For example, if either the upper or left pixel were alsoassigned a value of one, then that particular pixel would already belabeled as a segment by labeling sub-array 44. Accordingly, labelingsub-array 44 would label the subject pixel with the same segment numberas already applied to its neighboring pixel. If the upper and leftpixels are labeled differently, the left pixel would then be re-labeledto match the upper, or first labeled, pixel. Pixels within a connectedsegment are thus labeled in accordance with that particular segmentnumber by labeling sub-array 44. This process is continued for eachpixel in array 35. Clearly, however, other processes for analyzing andlabeling neighboring pixels may be performed without affecting the scopeof the present invention. Furthermore, although labeling algorithm 100is described as analyzing and labeling segments which include onlypixels which have adjacent or connected sides, other algorithms may beimplemented which label segments which have pixels adjacent at theircorners or within a predetermined range and/or intensity of each other.

After the three pixel windows 43 and 44 have completed analyzing andlabeling each of the pixels 42 within the imaging arrays, each of thediscreet segments are grouped together and labeled numerically, as shownin FIGS. 8A and 8B for imaging array sensors 34 a and 34 b,respectively. The average pixel location and maximum intensity of eachsegment may then be determined in order to facilitate a comparison ofthe segments on their respective sensors. This is accomplished bysumming the x and y pixel coordinates for the pixels within each segmentand dividing each sum by the number of pixels within the segment. Forexample, segment number (2) in FIG. 8A would have an average x positionof 5.67 $( \frac{5 + 6 + 6}{3} )$from a left edge 35 c of array 35 a and an average y position of 2.67$( \frac{2 + 3 + 3}{3} )$from an upper edge 35 d of array 35 b. Because the two imaging sensors34 a and 34 b are separated by a predetermined distance, each of theparticular segments representing a particular light source may bepositioned differently on imaging array sensor 34 b as compared to acorresponding segment on the other imaging array sensor 34 a, dependingon the distance and lateral orientation between the sensors and thelight source in the targeted scene. This is represented in FIG. 8B,where segment number (2) is received by sensor 34 b such that it has anaverage x position of 6.67 $( \frac{6 + 7 + 7}{3} )$and the same average y position as the segment had on the sensor 34 a inFIG. 8A. The distance may then be calculated using equation (1) above,where X_(D1) and X_(D2) are the directed distances from a referencepoint or center axis 34 c and 34 f of each sensor 34 a and 34 b to theaverage position of the particular segment on each sensor. In thisexample, X_(D1) may be a distance corresponding to separation of 1.67pixels while X_(D2) may be a distance corresponding to a separation of2.67 pixels, with the center axes 34 c and 34 f being at the center ofthe depicted arrays. Vehicle imaging system 12 may then determine if theintensity and location of the segments are consistent with the relevantor targeted images or light sources, such as headlamps or taillights,and may display an image or readout or adjust an associated accessory ofvehicle 10 accordingly.

Although described as preferably utilizing segmentation and averagingalgorithms, the present invention may alternatively compare individualpixels on one array to similarly illuminated individual pixels on theother array. Because the preferred embodiment groups similarlyclassified and positioned pixels together into segments and determines amaximum intensity and average location of the segment, the preferredsystem provides improved accuracy for distance calculation over acomparison of individual pixels. This is because the measurementresolution is then not limited to a pixel separation distance, since theaverage or center location of the sensed light source may be somewherebetween two or more pixels. Accordingly, the preferred control of thepresent invention provides sub-pixel resolution in the distancecalculation.

Referring now to FIG. 9, labeling algorithm or process 100 determinesand labels the segments of similarly classified pixels on each imagingarray sensor. Process 100 starts at 110 and compares each individualpixel to at least two neighboring pixels. If it is determined at 120that the target pixel has not been assigned a value of one, or is notabove a threshold value, then process 100 moves to the next pixel at 125and continues at 115. If it is determined at 120 that the target pixelvalue is greater than the threshold value or, in other words, has beenassigned a value of one, then it is further determined at 130 whetherthe pixel value is greater than the values associated with both an upperadjacent pixel and left adjacent pixel. If it is determined at 130 thatthe pixel value is greater than both of the upper and left pixels, thenthat particular pixel is assigned a new segment number at 135 andprocess 100 moves to the next pixel at 125 and continues at 115. If itis determined at 130 that the pixel value is not greater than both theupper and left pixel, then it is further determined at 140 whether thepixel value is equal to the upper pixel and not equal to the left value.If the pixel value is equal to the upper pixel and is not equal to or isgreater than the left pixel, then the particular pixel is assigned thesame segment number as the upper pixel at 145 and the process 100 movesto the next pixel at 125 and continues at 115.

If it is determined at 140 that the pixel value is not equal to theupper pixel or is equal to the left pixel, then it is further determinedat 150 whether the pixel value is both equal to the left pixel and isnot equal to or is greater than the upper pixel. If it is determined at150 that the pixel value is equal to the left pixel and is not equal tothe upper pixel, then the particular pixel is assigned the same segmentnumber as the left pixel at 155, and process 100 moves to the next pixelat 125 and continues at 115. If it is determined at 150 that the pixelvalue is not equal to the left pixel value or is equal to the upperpixel value, then it is further determined at 160 whether the pixelvalue is equal to both the left and upper pixels and the left and upperpixels are labeled the same. If it is determined at 160 that the pixelvalue is equal to the left and upper assigned values and the left andupper pixels are labeled the same, then the particular pixel is labeledthe same as the upper pixel at 165. Process 100 then moves to the nextpixel at 125 and continues at 115. If, however, the left label is notequal to the upper label at 160, then the particular pixel is labeledthe same as the upper pixel and the left pixel is correspondinglyrelabeled to the same as the upper pixel at 170, since the target pixelnow joins the left and upper pixel within the same segment. Process 100then moves to the next pixel to 125 and continues at 115 until eachpixel within each imaging array sensor has been analyzed and labeledaccordingly. Process 100 may be performed one or more times on each ofthe pixelated imaging array sensors in order to provide optimal results.

After labeling process 100 has been performed on each of the pixelatedimaging array sensors 34 a and 34 b, the pixels are labeled according tothe segments or groups of pixels associated with particularly classifiedlight sources. Once each particular segment is labeled on each sensor,additional algorithms or processes may be performed by control 16, inorder to determine a location and intensity of each segment with respectto the particular sensor. As shown in FIG. 10, a position and intensityprocess 200 determines an average x and y position of each segmentrelative to its respective sensor and a maximum intensity associatedwith each segment. Process 200 analyzes each pixel in each array andstarts at 210. Process 200 sets each position and intensity value foreach segment to zero at 220. If it is determined at 230 that the labelfor the pixel being analyzed is not equal to one of the previouslydesignated segment numbers, then process 200 moves to the next pixel at235 and continues at 237. If, on the other hand, the label associatedwith the particular pixel is equal to one of the segment numbers, thenthe x position and y position values for that segment are summed at 240.The x position value for the particular segment is the sum of thepreviously calculated x position value for that segment plus the xordinate for the particular pixel relative to the sensor array. The yposition value for that segment is similarly calculated and a countervalue is increased by one at 240.

It is then determined at 250 whether an image intensity value for thatpixel is greater than the maximum intensity value associated with thatparticular segment. If the pixel intensity value is greater than themaximum intensity for that segment, then the maximum intensity value forthat segment is set to the sensed image intensity value for theparticular pixel at 260. It is then determined at 270 whether all thepixels on each array have been analyzed. If it is determined at 270 thatnot all the pixels have been analyzed, then process 200 moves to thenext pixel at 235 and continues at 237. If it is determined at 270 thatthe pixels have all been analyzed, then an average x position and yposition associated with each segment is then calculated at 280 bydividing the summed x and y position values for each segment by thecorresponding count value for each particular segment. The process endsat 290. Upon completion of process 200, an average x and y position anda maximum intensity associated with each segment is stored forcomparison with the positions and intensities sensed by the other arraysensor. The positional values may be converted to conventional units ofmeasurement for use in the distance calculations of equation (1).

Referring now to FIG. 11, a distance algorithm or process 300 comparesthe average positions and intensities of each segment to correspondingsegments on the other sensor 34 b in order to determine whether asegment on the first sensor 34 a represents the same object or lightsource as a corresponding segment on the second sensor 34 b. Process 300begins at 310 and selects a first segment at 320. If it is determined at330 that an average x position and y position of the segment on thefirst sensor is within a predetermined position threshold of the averagex position and y position of a segment on the second sensor, then it isfurther determined at 340 whether the maximum intensities associatedwith each segment on each sensor are within a maximum intensitythreshold. If the average x and y positions are not within the positionthreshold at 330, then the process 300 moves to the next segment at 333and continues at 335. Likewise, if the maximum intensities are notwithin the maximum intensity threshold at 340, the process moves to thenext segment at 333 and continues at 335. If the average x and ypositions are within the position threshold at 330 and the maximumintensities are within the maximum intensity threshold at 340, adistance to that object or light source is calculated at 350, preferablyas a function of the x positions of the sensed light source on bothsensors according to equation (1), discussed above.

Because the vehicle imaging system 12 of the present inventionpreferably adjusts or controls an accessory of vehicle 10 in response tothe closest object or light source sensed by sensors 34 a and 34 b, itmay also be determined at 360 whether the calculated distance is lessthan a lowest distance for all segments. This provides the system withthe distance to the closest object or light source that has beenclassified by control 16. If it is determined at 360 that the distanceis less than a lowest distance value, then the lowest distance value isset to the newly calculated distant value at 370. It is then determinedat 380 whether all the segments have been accounted for. If it isdetermined at 380 that not all the segments have been accounted for, theprocess moves to the next segment at 333 and continues at 335. If, onthe other hand, it is determined at 380 that all the segments have beenaccounted for, the process ends at 390. Upon completion of process 300,the least distance from the vehicle 10 to a sensed object or lightsource which is in a selected classification and within a position andmaximum intensity threshold is stored for use by the imaging control 16.Control 16 may then function to display a distance readout or adjust theappropriate accessory of vehicle 10 in response to the intensity of thelight source sensed and/or the calculated distance to that light source.Algorithms 100, 200 and 300 may then be repeated for differentclassifications of light sources. For example, segments may beclassified as white or red light sources for headlamps or taillights orany other color which may be of interest to an operator of the vehicle.

Referring now to FIG. 12, a process 500 is shown which calculates adistance from an imaging array sensor or sensors to an object or lightsource sensed by the sensors and provides an output signal in responseto the distance and intensity of the light source. The output signal maybe in the form of a distance display or may provide an activation signalto a control, depending on the particular application of thestereoscopic imaging process 500. Process 500 begins at 505 and grabs acolor frame in each sensor or camera at 510 and 512. each pixel is thenclassified according to a desired color or other characteristic at 520and 522. The classified pixels are assigned a value of one, while theremaining pixels are assigned a value of zero and a segment labelingalgorithm similar to process 100 discussed above is performed at 530 and532 for the respective sensors. Clearly, however, the classified pixelsmay be designated in other manners, without affecting the scope of thepresent invention. The average x and y pixel locations and maximumintensity of each segment are then determined at 540 and 542. Process500 then compares the segmented images from both sensors at 550 andcalculates the distance to the light source corresponding to the similarsegments in both sensors at 560. The angular or lateral position of theobject or light source may also be determined at 560. It may then bedetermined at 570 if the distance and maximum intensity of a particularsegment are within a predetermined threshold. If the distance andmaximum intensity are within the threshold levels, then an appropriateoutput signal is sent at 580 and the process continues at 590. If, onthe other hand, the distance and/or maximum intensity are not within thethreshold at 570, then the process may continue at 590.

Although shown in FIG. 3 as having sensors 34 a and 34 b and lenses 36 aand 36 b positioned such that their optic paths are substantiallyparallel, clearly other orientations are within the scope of the presentinvention. For example, as shown in FIG. 13A, two oppositely facingsensors 34 a and 34 b may be implemented within a housing 29 or the likesuch that a pair of flat reflective surfaces or mirrors 37 a and 37 bare positioned along the respective optic paths between the lenses 36 aand 36 b and the sensors 34 a and 34 b. Alternately, a pair of openings39 a and 39 b may be provided in the housing 29 to allow light to passtherethrough such that it is redirected by the flat reflective surfaces37 a and 37 b toward the respective sensors 34 a and 34 b. The focusinglenses 36 a and 36 b may then be positioned along the respective opticpaths between the flat reflective surfaces 37 a and 37 b and the sensors34 a and 34 b (FIG. 13B). In another alternate orientation, a singleimaging array sensor 34 may be implemented within housing 29 to receivea stereoscopic image of the scene remote from the vehicle. A divider 41may be implemented substantially adjacent to sensor 34 to divide sensor34 into separate and distinct sensing arrays 34 a′ and 34 b′ (FIG. 13C).An additional pair of flat reflective surfaces or mirrors 42 a and 42 bmay also be included to redirect the image rays toward sensor 34 viafocusing lenses 36 a and 36 b. Clearly, however, the scope of thepresent invention includes other orientations where the lenses and oneor more reflective surfaces may be implemented along an optic pathbetween one or more sensors and the target scene.

Although vehicle imaging system 12 is useful in various imaging systemapplications, the control is particularly useful with a vehicle headlampdimming control 12′ (FIGS. 14 and 15). Vehicle headlamp control 12′ maythen classify the pixels as red, white or black and correspondinglyidentify the light sources as taillights or headlamps, using theprinciples disclosed in commonly assigned U.S. Pat. No. 5,796,094,referenced above. Headlamp control 12′ may determine the distancesbetween vehicle 10 and the identified taillights and headlamps andcommunicate this information to a vehicle lighting control logic module18 (FIG. 15). Vehicle lighting control logic module 18 may then exchangedata with control 16 to control headlamps 20 of vehicle 10 in responseto the output of sensor module 14 as received by imaging control 16.Imaging control 16 may analyze detected light sources to determine acolor and/or intensity of the light sources and to determine a distancebetween the light sources and vehicle 10. This information may then becommunicated to lighting control logic module 18 for dimming ofheadlamps 20. Dimmer control 12′ thus may correspondingly control theheadlamps 20 in response to the color or intensity of the light sourcesas well as the distance to the light sources. Additional criteria mayalso be considered, such as the lateral position of the sensed lightsources with respect to the vehicle or other criteria associated withsize, color, position, intensity or rate of approach of the lightsource.

Preferably, as shown in FIG. 14, imaging sensor module 14 may be fixedlymounted in housing 28 by a bracket 24 mounted to, or near, the vehicle'swindshield 26. Sensor module 14 may be mounted within housing 28 invarious orientations, as discussed above with respect to FIGS. 13A-13C.Bracket 24 may also mount an interior rear-view mirror 30. However,imaging sensor module 14 may be mounted elsewhere on the vehicle withoutaffecting the scope of the present invention.

Referring now to FIGS. 16A and 16B, a headlamp control process 400 isshown which starts at 405 by determining whether the ambient light levelis below a predetermined threshold. If the light level is below thethreshold, then process 400 grabs a color frame at a headlamp shuttersetting for both cameras or sensors 34 a and 34 b at 410 and 412,respectively. Process 400 then classifies each pixel as white or blackat 415 and 417 and assigns a value of one to white pixels and a value ofzero to black pixels at 420 and 422 or otherwise designates the pixels.The segment labeling algorithm 100 is performed at 420 and 422 for thetwo sensors 34 a and 34 b, respectively. An average x and y pixellocation and maximum intensity is then calculated according to process200 at 425 and 427 for each segment on the respective sensors. Headlampcontrol process 400 then compares the location and intensity of thesegmented images from both sensors at 430 in order to determine segmentson each sensor which correspond to a particular light source. Controlprocess 400 determines that the segments correspond to a particularlight source if the compared segments on both sensors are within an x-ypixel space threshold and intensity threshold, in accordance withprocess 300, discussed above. The distance to the light sourcecorresponding to the similar segments is then calculated at 440. Theangular and/or lateral position of the light source relative to vehicle10 may also be calculated at 440. It is then determined at 450 whetherthe distance and maximum intensity of corresponding segments areconsistent with a headlamp of an oncoming vehicle and within apredetermined threshold level. The consistency criteria may include aforward and lateral position relative to vehicle 10, intensity, size, orany other criteria which may discern a headlamp form other lightsources, such as rate of approach or the like relative to vehicle 10. Ifit is determined at 450 that the distance, intensity and/or any otherselected criteria are within the threshold levels, the headlamps are setto a low beam setting at 452 and the process returns at 455.

If it is determined at 450 that the distance, maximum intensity or othercharacteristics of the segment are not consistent with a headlamp orwithin the threshold level, then process 400 grabs color frames at ataillamp shutter setting in camera sensors 34 a and 34 b at 460 and 462,respectively, using the principles disclosed in U.S. Pat. No. 5,796,094,referenced above. Each pixel is then classified as red or black at 465and 467. The red pixels are then assigned a value of one or otherwisedesignated, while the black pixels are assigned a value of zero orotherwise designated, at 470 and 472. The segment labeling algorithm 100is again performed on each of the respective sensors at 470 and 472. Anaverage x and y pixel location and maximum intensity are then calculatedaccording to process 200 at 475 and 477 for each segment on therespective sensors. The segmented images from both cameras are thencompared at 480 to determine which segments are close in x-y pixelpositioning and similar in maximum intensity between the two sensors.The distance to a light source corresponding to the similar segments inboth sensors is then calculated at 485. The lateral position of thelight sources may also be determined at 485. It is then determined at490 if the distance and maximum intensity of the segment are consistentwith a taillamp and within a predetermined threshold. Similar to theconsistency criteria above with respect to headlamps, the light sourcemay be analyzed to determine if their size, intensity, lateral andvertical position relative to vehicle 10 and/or rate of approach tovehicle 10 are consistent with known or assumed values associated withvehicle taillights. If the distance, maximum intensity and the like arewithin the threshold levels, the headlamps are set to a low beam at 492and the process returns to 405 at 455. If, on the other hand, thedistance, maximum intensity and/or other selected criteria are notconsistent with taillamps or are not within the threshold levels, theheadlamps are set to a high beam setting at 495 and the process againreturns at 455. Process 400 thus adjusts the headlamp setting inresponse to the distance and maximum intensity of light sources sensedby both of the sensors 34 a and 34 b.

The present invention thus accounts for both the intensity of lightsensed by the sensors and the distance to the light source from thevehicle 10, before adjusting the headlamp setting for the vehicle. Thisallows the vehicle headlamps to remain in a high beam setting untilvehicle 10 is within a predetermined range of a sensed headlamp ortaillight, and conversely, the headlamps may be set to a high beamsetting once a sensed headlamp or taillight moves beyond thatpredetermined range. By sampling real World data or simulating variousdriving conditions, a pixel intensity versus distance curve may becreated which is typical of headlamps and taillamps for various drivingconditions. Such a curve is shown in FIG. 17A, where a segment intensityand corresponding distance at point A below the curve would not beclassified as a headlamp, while a signal B, which has similar intensitybut greater distance than point A, may be classified as a headlamp.Headlamp control process 400 is then further optimized since certainsegments 110 which are not within a range of the real world data curvewould not be included in the headlamp analysis. Similarly, as shown inFIG. 17B, real world data may be used to modify the curve such that anangular position of the light source relative to vehicle 10 is furtherincluded in the analysis in order determine whether or not the segmentshould be classified as a headlamp or taillight. For example, the signalC in FIG. 17B would be classified as a headlamp if it is determined tobe at approximately a 15° angle relative to vehicle 10, but may not beclassified as a headlamp if it is only approximately 0°-5° off of theaxis of the sensors 34 a and 34 b in vehicle 10. The system may beotherwise optimized as shown if FIG. 17C, where a minimum and maximumpixel intensity band 60 versus distance is implemented. With such aband, segments which fall within the shaded area or band 60, such aspoint D, may be classified as headlamps, while segments falling outsideof the band 60, such as points E and F, may not be classified asheadlamps by headlamp control process 400. Clearly, the scope of thepresent invention further includes other thresholds and criteria fordetermining whether a particular segment should be classified as aheadlamp or taillight, with respect to its intensity and distance and/orangle or lateral position relative to vehicle 10.

Therefore, the present invention provides a stereoscopic imaging systemuseful with various accessory controls or displays which is operable todetermine a distance from one or more imaging array sensors to an objector light source remote from the sensors. The stereoscopic imaging systemmay determine a distance to any object or light source in a targetedscene, without requiring additional equipment or ranging devices.Furthermore, the system may provide a distance determination to aheadlamp control, without having to assume that the light source iswithin a predetermined range of intensities corresponding to a typicalintensity of a headlamp or taillight and calculating the distance basedon the intensity alone. Accordingly, the imaging system provides a moreaccurate distance calculation, since it is not affected by variations inthe intensity of the light source that is being sensed. The accuracy ofthe distance calculations may be further enhanced by implementing asegmentation algorithm which determines the average position of thelight source as received by the sensor, thereby facilitating sub-pixelresolution for the distance calculations. Furthermore, the distancecalculation may be applied equally as well to other images that are notassociated with headlamps or taillights of other vehicles. Accordingly,the stereoscopic imaging system described herein may be useful withother vehicular imaging systems, such as rearview vision systems, backupaids, rain sensors or the like.

Changes and modifications in the specifically described embodiments canbe carried out without departing from the principles of the invention,which is intended to be limited only by the scope of the appendedclaims, as interpreted according to the principles of patent law.

1-98. (canceled)
 99. A vehicular imaging system suitable for use in avehicle, said vehicular imaging system comprising: an imaging arraysensor having a field of view directed outwardly from the vehicle andtoward an external scene, said imaging array sensor having at least afirst portion and a second portion, said first portion capturing a firstimage of the external scene and said second portion capturing a secondimage of the external scene; a first optic element and a second opticelement, said first optic element positioned along a first optic pathbetween said first portion of said imaging array sensor and the externalscene, said second optic element positioned along a second optic pathbetween said second portion of said imaging array sensor and theexternal scene; and a control responsive to an output of said imagingarray sensor, said control processing said output and distinguishingsaid first image and said second image in order to identify objects ofinterest in the external scene, said control generating a control outputin response to said processing.
 100. The imaging system of claim 99,wherein said control comprises at least one of a headlamp control, areverse aid imaging system control and a rain sensor control.
 101. Theimaging system of claim 99, wherein said control comprises a headlampcontrol.
 102. The imaging system of claim 101, wherein said controladjusts a headlamp setting in response to said processing.
 103. Theimaging system of claim 102, wherein said imaging array sensor has aforward field of view.
 104. The imaging system of claim 103, whereinsaid imaging array sensor has a forward field of view through thewindshield of the vehicle.
 105. The imaging system of claim 99, whereinsaid control comprises a reverse aid imaging system.
 106. The imagingsystem of claim 105, wherein said control controls at least one of adisplay, an alert device and said imaging array sensor in response tosaid processing.
 107. The imaging system of claim 106, wherein saidimaging array sensor has a rearward field of view toward an externalscene rearward of the vehicle.
 108. The imaging system of claim 99,wherein said control comprises a rain sensor control.
 109. The imagingsystem of claim 108, wherein said control controls at least one of awindshield wiper of the vehicle and a blower motor of a heating,ventilation and air conditioning system of the vehicle in response tosaid processing.
 110. The imaging system of claim 99, wherein saidcontrol is operable to control at least one accessory of the vehicle inresponse to said processing.
 111. The imaging system of claim 110,wherein said at least one accessory comprises at least one of a displaysystem of the vehicle, a warning indicator of the vehicle, a signalingdevice of the vehicle, a collision avoidance system of the vehicle, abrake system of the vehicle, a headlamp of the vehicle, a windshieldwiper of the vehicle, and a blower motor of a heating, ventilation andair conditioning system of the vehicle.
 112. The imaging system of claim110, wherein said at least one accessory comprises a display system ofthe vehicle.
 113. The imaging system of claim 112, wherein said controlis operable to adjust a display of said display system in response to adistance to a closest object of interest.
 114. The imaging system ofclaim 113, wherein said control is operable to adjust said display todisplay the distance to the closest object of interest.
 115. The imagingsystem of claim 113, wherein said control is operable to adjust saiddisplay of said display system in response to at least one of aclassification of the closest object of interest, an intensity of theclosest object of interest, a size of the closest object of interest,and a position of the closest object of interest.
 116. The imagingsystem of claim 99, wherein said imaging array sensor comprises at leastone pixelated imaging array sensor comprising a plurality of pixels.117. The imaging system of claim 99, wherein said first optic has afirst focal length and said second optic has a second focal length. 118.The imaging system of claim 117, wherein said control generates saidcontrol output as a function of said first and second focal lengths.119. The imaging system of claim 99, wherein said control is operable todetermine a distance and angle to a detected object of interest.